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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )S Responsive to communication(s) filed on 07 June 2004 . 
2a)D This action is FINAL. 2b)^ This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 1-17 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) KI Claim(s) 1-17 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121 (d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2.0 Certified copies of the priority documents have been received in Application No. . 



3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 



1. This action is responsive to communications: request for continued examination filed 
6/7/2004, declaration of ownership and assignment for application 09/410,781 filed on 6/7/2004. 

2. The rejection of the present application under 35 U.S.C. 103(a) as being unpatentable 
over Srivastava et al., US 6,549,922 Bl filed 10/1/1999 in view of Sidana, US 6,571,295 Bl 
priority filed 1/31/1996 is withdrawn in response to the Declaration of Ownership submitted on 



3. Claims 1-17 are pending in the case. Claims 1, 8, and 10 are independent claims. 



4. The Examiner accepts the Declaration of Ownership for application 09/410,781 now US 
patent 6,549,922 Bl and notes the inventors were working for Oracle Corporation on the 
10/1/1999 date of filing and had agreed to assign inventions made during their course of 
employment to Oracle Corporation. Consequently, the present application and Srivastava et al. 
were co-owned by Oracle Corporation on 2/1/2000 thus precluding the use of Srivastava et al. as 
prior art against the present application. The rejection of the present application under 35 U.S.C. 
103(a) as being unpatentable over Srivastava et al., US 6,549,922 Bl filed 10/1/1999 in view of 
Sidana, US 6,571,295 Bl priority filed 1/31/1996 is withdrawn. 



6/7/2004. 



Oath/Declaration 



Claim Rejections - 35 USC § 103 
5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

6. Claims 1-2, 4-8, and 10-17 are rejected under 35 ILS.C. 103(a) as being unpatentable 
over Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998. 

Regarding independent claim 1, Nelson teaches analyzing a multimedia document to 
identify at least one markup tag containing a reference to a given one of resources in fig. 1-2, fig. 
4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 line 7. Nelson shows in the col. 
5 lines 39-5 1 that the multimedia document can be an HTML document and HTML documents 
generally represent Web pages, thus the scope of Nelson includes multimedia Web page 
documents. Nelson also specifically indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the 
multimedia document may be a downloaded Web page. Nelson teaches selecting and executing 
a media processing program for analyzing the content of the multimedia data supplied by the 
given one of the resources to generate metadata describing the content in fig. 2, fig. 4-7, and col. 
9 line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for 
each of the types of multimedia content components. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 
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Nelson does not teach combining the Web page and the generated annotation (Nelson 
calls it a token) to form an enhanced Web page. As stated earlier, Nelson does teach that all of 
the tokens extracted from a particular document are linked to that document through use of a 
unique document ID in col. 15 lines 24-36. Therefore, tokens of text, image, video, audio, and 
other data are all linked together to represent a single document through use of the unique 
document ID. Thus, this is essentially the same in function, but only similar in form to the 
claimed limitation. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have modified Nelson to have created the claimed invention. It would 
have been obvious and desirable to have modified Nelson to have inserted the token 
representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 2, Nelson teaches determining the particular data type of 
the multimedia data supplied by the given resource and selecting a processing program for 
analyzing multimedia data formatted in accordance with the particular data type in fig. 2, fig. 4, 
and col. 3 lines 21-27. 

Regarding dependent claim 4, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. 

Regarding dependent claim 5, Nelson teaches wherein at least some of the additional 
data includes information obtained from the one markup tag in fig. 2, fig. 4-7, and col. 9 line 9 - 
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col. 14 line 55. Nelson indicates in col. 9 line 9 - col. 10 line 37 that the tags may be used to 
identify a position and a name for the identified multimedia component. 

Regarding dependent claim 6, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson does not specifically teach wherein the given resource is accessed 
through the operating system of the computer which provides the given resource and wherein at 
least some of the additional data includes information obtained from the operating system. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have modified Nelson to have created the claimed invention. Nelson teaches the 
multimedia content components may be referenced from the multimedia document and thus a 
valid reference would be a multimedia content component from the operating system. It would 
have be obvious and desirable to have obtained a referenced resource from the operating system 
and used metadata from the operating system to describe the resource so that multimedia 
components could have taken advantage of the local computer's operating system resources. 

Regarding dependent claim 7, Nelson teaches acquiring additional metadata which 
describes the multimedia data supplied by the given one of the resources, and including the 
additional metadata in the character-based text annotation in fig. 2, fig. 4-7, and col. 9 line 9 - 
col. 14 line 55. Nelson teaches in col. 5 lines 39-51 that the multimedia component may be 
referenced by tags in the multimedia document. If the reference indicates a resource located on 
the Internet, Nelson is then going to obtain additional information about the resource via the 
Internet in forming the characterization token. 
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Regarding independent claim 8, Nelson teaches a parser for identifying markup tags in 
a Web page in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 line 
7. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches analyzing the content of multimedia data identified by markup tags to generate metadata 
describing the multimedia data in fig. 2, fig. 4-7, and col. 9 line 9 - col. 14 line 55. The cited 
text section described how Nelson generates metadata for each of the types of multimedia 
content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach storing the combination of a copy of the Web page and the 
annotation to form an enhanced Web page suitable for processing by text-based indexing and 
searching facilities. As stated earlier, Nelson does teach that all of the tokens extracted from a 
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particular document are linked to that document through use of a unique document ID in col. 15 
lines 24-36. Therefore, tokens of text, image, video, audio, and other data are all linked together 
to represent a single document through use of the unique document ID. Thus, this is essentially 
the same in function, but only similar in form to the claimed limitation. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to have modified 
Nelson to have created the claimed invention. It would have been obvious and desirable to have 
modified Nelson to have inserted the token representations of image, video, audio, and other 
multimedia components into the text of the multimedia document so that the text did not have to 
be processed, thus allowing the extraction software design to be simpler. 

Regarding independent claim 10, Nelson teaches identifying one or more markup tags 
in a Web page which respectively identify one or more external resources which provide 
multimedia data in fig. 1-2, fig. 4, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 9 
line 7. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches generating metadata which describes the multimedia data in fig. 2, fig. 4-7, and col. 9 
line 9 - col. 14 line 55. The cited text section described how Nelson generates metadata for each 
of the types of multimedia content components. 

Nelson teaches translating the metadata into a character-based text annotation describing 
the multimedia data in fig. 6-8 and col. 3 lines 19-59. Nelson teaches formatting the metadata 
into character-based tokens in fig. 6-8 and col. 3 lines 19-59. The tokens for the different 
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multimedia component types are normalized so that tokens can be combined into a single index 
as is shown in fig. 8. Nelson teaches indexing the multimedia document into a unified 
multimedia index to equally represent in a normalized fashion all of the tokens of the multimedia 
document in fig. 8-9 and col. 14 line 56 - col. 15 line 58. Each of the tokens is linked to the 
document from which it was extracted through use of a unique document ID referencing that 
document. 

Nelson does not teach inserting the annotation into the Web page to form an enhanced 
Web page suitable for processing by a character-based text processing system. As stated earlier, 
Nelson does teach that all of the tokens extracted from a particular document are linked to that 
document through use of a unique document ID in col. 15 lines 24-36. Therefore, tokens of text, 
image, video, audio, and other data are all linked together to represent a single document through 
use of the unique document ID. Thus, this is essentially the same in function, but only similar in 
form to the claimed limitation. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to have modified Nelson to have created the claimed invention. 
It would have been obvious and desirable to have modified Nelson to have inserted the token 
representations of image, video, audio, and other multimedia components into the text of the 
multimedia document so that the text did not have to be processed, thus allowing the extraction 
software design to be simpler. 

Regarding dependent claim 11, Nelson teaches first identifying markup tags in a Web 
page and extracting the uniform resource locator (URL) of one of the external resources from the 
at least selected ones of the markup tags in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, 
and col. 8 line 1 - col. 14 line 55. Nelson shows in the col. 5 lines 39-51 that the multimedia 



Application/Control Number: 09/496,086 Page 9 

Art Unit: 2176 

document can be an HTML document and HTML documents generally represent Web pages, 
thus the scope of Nelson includes multimedia Web page documents. Nelson also specifically 
indicates in col. 5 lines 31-38 and col. 8 lines 26-27 that the multimedia document may be a 
downloaded Web page. Nelson teaches in col. 5 lines 39-51 that the multimedia component may 
be referenced by tags in the multimedia document. The reference to a multimedia component in 
an HTML Web page is a URL. 

Regarding dependent claim 12, Nelson teaches retrieving multimedia data from one or 
more external resources and analyzing the content of the multimedia data to extract the metadata 
therefrom in fig. 1-2, fig. 4-7, col. 2 lines 19-37, col. 5 lines 11-51, and col. 8 line 1 - col. 14 line 
55. Nelson shows in the col. 5 lines 39-51 that the multimedia document can be an HTML 
document and HTML documents generally represent Web pages, thus the scope of Nelson 
includes multimedia Web page documents. Nelson also specifically indicates in col. 5 lines 31- 
38 and col. 8 lines 26-27 that the multimedia document may be a downloaded Web page. Nelson 
teaches in col. 5 lines 39-51 that the multimedia component may be referenced by tags in the 
multimedia document. The reference is to an external resource from which the multimedia data 
is retrieved and the metadata extract therefrom. 

Regarding dependent claim 13, Nelson teaches identifying the data type of the 
multimedia data from each of the resources and then selecting a processing routine for 
multimedia of the identified data type from each of the resources in fig. 2, fig. 4, and col. 3 lines 
21-27. 

Regarding dependent claim 14, Nelson teaches indexing the multimedia document into 
a unified multimedia index to equally represent in a normalized fashion all of the tokens of the 
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multimedia document and provide access to the Web page in response to queries expressing one 
or more attributes expressed in the text annotation in fig. 8-9 and col. 14 line 56 - col. 15 line 58. 
Each of the tokens is linked to the document from which it was extracted through use of a unique 
document ID referencing that document. 

Regarding dependent claim 15, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 
contained in the text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

Regarding dependent claim 16, Nelson teaches indexing the Web page to provide 
access to the Web page in response to queries expressing one or more attributes expressed in the 
text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - col. 27 line 3. 

Regarding dependent claim 17, Nelson teaches searching the content of the Web page 
in response to a search request to determine if attributes expressed in the search request are 
contained in the text annotation in fig. 11-16, col. 3 line 47 - col. 4 line 17, and col. 15 line 60 - 
col. 27 line 3. 

7. Claims 3 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nelson et al. (hereinafter "Nelson"), US 6,243,713 Bl filed 8/24/1998 in view of Mohan et al. 
(hereinafter "Mohan"), US 6,748,382 Bl filed 12/1/1999 with priority to provisional filed 
1/28/1999. 

Regarding dependent claim 3, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
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text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 

Regarding dependent claim 9, Nelson does not teach generating a text data annotation 
expressed in accordance with the Extensible Markup Language. Mohan does teach generating a 
text data annotation expressed in accordance with the Extensible Markup Language in fig. 3 and 
col. 2 lines 16-28. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Mohan into Nelson to have created the claimed invention. 
It would have been obvious and desirable to have enhanced the tokens of Nelson with the XML 
based media asset management descriptions of Mohan to have ensured maximum 
interoperability as described by Mohan in col. 2 lines 27-28. 

Response to Arguments 
8. Applicant's arguments, see page 2, filed 6/7/2004, with respect to the rejection(s)of 
claim(s) 1-17 under 35 U.S.C. 103(a) as being unpatentable over Srivastava et al M US 6,549,922 
Bl filed 10/1/1999 in view of Sidana, US 6,571,295 Bl priority filed 1/31/1996 have been fully 
considered and are persuasive. The Declaration of Ownership proves that the present application 
and the cited patent were commonly owned by Oracle Corporation on the 2/1/2000 date of filing 
of the present application. Therefore, the rejection has been withdrawn. However, upon further 
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consideration, anew ground(s) of rejection is made in view of Nelson et al., US 6,243,713 Bl 
filed 8/24/1998 for claims 1-2, 4-8, and 10-17 and Nelson et al, US 6,243,713 Bl filed 
8/24/1998 in view of Mohan et al., US 6,748,382 Bl filed 12/1/1999 with priority to provisional 
filed 1/28/1999 for claims 3 and 9. 

Conclusion 

9. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Klemens Bohm & Thomas C. Rakow "Metadata for Multimedia Documents" 
SIGMOD Record, Vol. 23, No. 4, Dec. 1994 discloses different kinds of metadata for 
multimedia documents. Khosla et al., US 6,202,061 Bl filed 10/24/1997 discloses creating a 
collection of digital media. Logan, US 6,665,659 Bl filed 2/1/2000 discloses distributing and 
using metadata via the internet. 

10. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 703-305-593 1 . The 
examiner can normally be reached on Mondays-Fridays 7:00am-3:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Joseph H Feild can be reached on 703-305-9792. The fax phone number for the organization 
where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be 
obtained from either Private PAIR or Public PAIR. Status information for unpublished 
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applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

PJS 

August 26, 2004 




